Summary

• Why: resolveDCRCredentials was wired up and unit-tested in the prior sub-issue, but EmbeddedAuthServer never called it, so any upstream configured with DCRConfig still booted without a client. This PR makes the resolver load-bearing and adds the operator-visible structured logs the Phase 2 spec calls for (cache-hit age, stale-registration warning, per-step errors, and a remediation hint when an upstream rejects a cached client as 4xx).
• What:
  • EmbeddedAuthServer now owns an in-memory DCRCredentialStore and calls resolveDCRCredentials + consumeResolution for every OAuth2 upstream whose config requests DCR. buildPureOAuth2Config deliberately keeps its pure, network-free signature; the DCR-resolved ClientSecret is layered on afterwards via a new applyResolutionToOAuth2Config helper so the two application sites stay side-by-side.
  • Each upstream RunConfig element is shallow-copied and its OAuth2 sub-config is deep-copied before mutation, preserving the caller's RunConfig.Upstreams slice.
  • resolveDCRCredentials returns a new dcrStepError wrapping the originating step (metadata discovery, DCR call, store read, store write) instead of logging-and-returning at each branch. The boundary caller (buildUpstreamConfigs) emits one slog.Error via logDCRStepError, so there is a single error line per failure. A panic inside the singleflight closure is converted to the same dcrStepError carrying the captured debug.Stack(), so the boundary log surfaces it without an in-defer duplicate.
  • Cache-hit path emits slog.Debug with dcr_age_days and additionally slog.Warn with remediation text when the cached registration exceeds the 90-day dcrStaleAgeThreshold.
  • The /oauth/register handler success log emits at Debug with issuer, client_id, software_id, token_endpoint_auth_method, and scopes — silent at INFO+ per the project's logging convention. SoftwareID is threaded through ValidateDCRRequest, which now also caps it to 256 printable-ASCII characters.
  • MonitoredTokenSource accepts upstream and client_id at construction (replacing the earlier SetUpstreamContext setter, which had an unsynchronized writer). On the first transition to Unauthenticated driven by a permanent token-endpoint 4xx, it emits a single slog.Warn with a DCR/CIMD remediation hint (gated by stopOnce so a tight Token() loop cannot spam it, and suppressed when no client_id context is available). The classifier itself (isTransientNetworkError) is side-effect free.
  • NewHandler now fails loudly when AuthorizationServerConfig or its embedded *fosite.Config is nil; issuer() no longer carries a runtime nil-guard.
  • Error strings are run through a sanitizer that strips URL userinfo, query, and fragment components before logging. The regex matches http(s):// case-insensitively per RFC 3986 §3.1, and trailing sentence punctuation around a URL is preserved so prose is not mangled.
  • pkg/oauthproto/dcr.go handleHTTPResponse caps the upstream /register error body at 8 KiB via io.LimitReader so a non-conformant or malicious upstream cannot inflate operator log volume.

Closes #5039

Type of change

• New feature

Test plan

• Unit tests (task test)
• Linting (task lint-fix)

New coverage added:

• pkg/authserver/runner/embeddedauthserver_test.go — full DCR boot against a mock AS (metadata + /register), cache-hit short-circuit (zero additional HTTP requests on the second buildUpstreamConfigs), copy-before-mutate assertion on the caller's RunConfig.Upstreams slice, and TestNewEmbeddedAuthServer_DCRBoot driving the real constructor to assert dcrStore is populated.
• pkg/authserver/runner/dcr_test.go — cache-hit Debug log with dcr_age_days, stale-age Warn above threshold, one dcrStepError-returning case per failure step, and the panic-recovery test now asserts the wrapped error is a *dcrStepError(dcrStepRegister, …, Stack: <captured>).
• pkg/authserver/server/handlers/handler_chain_test.go — TestHandler_issuer covers both populated and zero-value AccessTokenIssuer. TestNewHandler_ErrorsOnNilConfig (in authorize_test.go) pins the new constructor invariant.
• pkg/authserver/server/registration/dcr_test.go — software_id length cap and non-printable-ASCII rejection.
• pkg/auth/monitored_token_source_test.go — permanent-4xx branch emits the remediation Warn with workload/upstream/client_id populated via the new constructor parameters.
• pkg/runner/runner_test.go — TestRemoteAuthLogContext covers the full cached CIMD > cached DCR > static precedence.
• Sanitizer unit tests cover commas, periods, closing parens, mixed runs, a Go http.Client-style quoted URL, embedded user:pass@host userinfo (with and without query), implicit-flow #access_token=… fragments, and uppercase HTTPS:// schemes.

Changes

Large PR Justification

The PR exceeds the 400-LOC / 10-file guideline (1342+/104- across 18 files), but the change cannot be split without leaving the codebase in an intermediate non-functional state:

• ~660 lines are tests (*_test.go) — table-driven coverage for the resolver wiring, the new error taxonomy, the staleness + remediation logs, and the sanitizer.
• The remaining ~510 production lines across 9 files implement one logical change (wire the resolver into the auth-server boot path) plus four tightly-coupled sub-concerns: resolver wiring, dcrStepError taxonomy + boundary logging, monitored-token-source remediation log, and software_id validation. Splitting the resolver wiring without the matching error taxonomy would land a regression in observability; splitting the error taxonomy without the wiring would orphan the new types.

Does this introduce a user-facing change?

Operators gain two new diagnostic signals on DCR-enabled upstreams:

1. A Warn when a cached DCR registration is older than 90 days, hinting that the operator may want to rotate.
2. A Warn on the first transition to Unauthenticated driven by a permanent token-endpoint 4xx, hinting that the workload's cached DCR or CIMD credentials may be stale and that deleting them and restarting will trigger re-registration. The Warn fires at most once per state transition and is suppressed entirely for workloads with no client_id context.

The MonitoredTokenSource constructor signature has changed to require upstream and client_id, but it is an internal API with no public surface, and the prior SetUpstreamContext setter is removed.

Successful DCR registrations log at Debug, not Info, in keeping with the project's silent-success-at-INFO+ convention.

Special notes for reviewers

• The purity gate on buildPureOAuth2Config is load-bearing: it takes no context.Context and performs no network I/O. DCR resolution is deliberately sequenced around it, not inside it, so the pure config builder stays trivially testable. Please flag any future change that threads ctx into it.
• dcrStepError moves the "one error line per failure" invariant from the resolver to the boundary caller. If a future caller forgets to invoke logDCRStepError, the error will still propagate normally but the structured log will be missing. Worth keeping in mind if we add more resolver call sites.
• consumeResolution (formerly applyResolution) is a one-shot state transition, not an idempotent default-fill: a second call after the first is a no-op only because the first cleared DCRConfig. The companion applyResolutionToOAuth2Config writes the inline-only ClientSecret onto the built *upstream.OAuth2Config; any future output path from OAuth2UpstreamRunConfig to upstream.OAuth2Config must call both helpers to get a fully-resolved DCR client.
• The package-level dcrFlight singleflight is intentionally process-wide while EmbeddedAuthServer.dcrStore is per-instance — see the doc comments on each. The asymmetry is load-bearing for concurrent EmbeddedAuthServer instances in the same process.

Generated with Claude Code